1. Field of the Invention
The present invention relates to an image processing technique to reconstruct various images using light ray information obtained by image capturing and output reconstructed images.
2. Description of the Related Art
Performing image capturing by introducing light from an imaging optical system to an image sensor through a microlens array or by using a combination of the imaging optical system and the image sensor (that is, an image pickup apparatus) can provide light ray information including intensities or directions of light rays from respective points in an object space. An image processing technique has been proposed which reconstructs, by using the light ray information, an image on which adjustment of a focus position, a depth of field or a viewpoint direction is performed. Ren Ng and seven others, “light field photography with a hand-held plenoptic camera”, Stanford Tech Report CTSR 2005-02A discloses a method “Light Field Photography” that acquires information on positions and angles (directions) of light rays from an object and reconstructs an image focused on an arbitrary position or an image viewed from an arbitrary viewpoint direction using the acquired information.
In general, image capturing using a wide-angle lens or image capturing using an image pickup apparatus with a small image sensor can provide, due to its short focal length, an image with a large depth of field (that is, an entirely in-focus image). In addition, image capturing with a narrowed aperture also can provide an image with a large depth of field. An image process (hereinafter referred to as “a blur addition process”) is sometimes performed on such images to intentionally add a blur (image blur component) thereto in order to emphasize a main object or soften atmosphere of the image.
Blurs appearing in images include a blur caused by defocus generated due to a position of an object whose distance is different from an in-focus distance and a blur caused by aberration or diffraction in an optical system. The above-mentioned blur which is intentionally added to the image is the blur caused by defocus (hereinafter simply referred to as “a blur” or “a defocus blur”).
Japanese Patent Laid-Open No. 2009-159357 discloses a technique which introduces light from an imaging optical system to an image sensor through a microlens array to obtain data of multiple pixels and performs weighting on the data corresponding to viewpoint directions to reconstruct an image in which a depth of field is adjusted or an image in which various blurs are expressed. Japanese Patent Laid-Open No. 2003-18438 discloses an image pickup apparatus which acquires images having different blurs added thereto at a same time using multiple cameras having imaging optical systems with different optical characteristics.
The technique disclosed in Japanese Patent Laid-Open No. 2009-159357 controls a shape of the blur in the reconstruction to express the various blurs. The technique disclosed in Japanese Patent Laid-Open No. 2003-18438 expresses the different blurs on a basis of image information obtained by the multiple cameras. However, in Japanese Patent Laid-Open Nos. 2009-159357 and 2003-18438, a size of the blur to be expressed is defined depending on an F-number of the imaging optical system, and a blur addition process as an image process to add to an image a blur whose size corresponds to an F-number smaller than the F-number of the imaging optical system is not disclosed.
Moreover, the blur addition process has the following problems. A first problem is caused by an optical characteristic of an imaging optical system. An image obtained by image capturing includes more than a little blur caused by the above-mentioned aberration or diffraction in the imaging optical system, regardless of a defocus amount, a size of an image sensor and a focal length. Adding the defocus blur to such an image including the blur caused by the aberration or diffraction makes it difficult to express a natural blur.
For example, when a defocus blur with a circularly symmetric shape is added to an image of an object as a point light source shown in FIG. 18A, a natural blur is obtained as shown in FIG. 18B. However, for example, when the circularly symmetric defocus blur is added to an image in which a blur occurs due to coma aberration as shown in FIG. 18C, an unnatural blur is obtained as shown in FIG. 18D.
A second problem arises when a blur is added to a luminance saturated area. FIG. 19A shows a luminance distribution of one section of the luminance saturated area. A saturation value in FIG. 19A is a maximum luminance value which can be expressed by a digital image. When a blur is added to the luminance saturated area shown in FIG. 19A, a gentle curve-shaped luminance distribution shown in FIG. 19B is obtained. FIG. 19C shows a luminance distribution of an image captured under a condition that an F-number is less than that shown in FIG. 19A. The luminance distribution of the blur shown in FIG. 19B is less than that shown in FIG. 19C. That is, it is impossible to express a blur in the luminance saturated area of the image captured under the condition that the F-number is small, only by adding the blur to the luminance saturated area.